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ABSTRACT 


The hypothesis is advanced that testing and evaluation 
(T&E) of complex weapons systems requires unique skills, that 
the testing and evaluation of weapons systems has evolved 
into a recognizable engineering discipline, and that pro- 
fessional technical personnel in the Department of Defense 
Test and Evaluation community should be considered as unique 
assets and supported by the establishment of a postgraduate 
curriculum in T&E engineering. 

The evolution of DOD T&E is traced and analyzed with 
particular attention to capability requirements of personnel. 

The general conclusion is reached that the hypothesis can 
not now be universally supported. Reasons for this position 
are given and recommendations made for improving capabilities 


of T&E personnel. 
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I. INTRODUCTION 


A. GENERAL 

Test and Evaluation (T&E) is conducted to support the 
acquisition of weapon systems by the Department of Defense 
(DOD). These weapon systems integrate the latest advances 
in technology and are complex. T&E field activity personnel 
have suggested that advanced education is required to support 
the T&E of modern weapon systems. It is also believed that 
T&E is evolving as an engineering discipline, and a graduate 


level curriculum in T&E engineering should be established. 


Sm OpIECTIVES 

The objectives of this study are to: 

1. Show that T&E is an important element of the weapon 
systems acquisition cycle and should be recognized as a 
discipline which is unique to DOD. 

2. Determine if there is a need for T&E postgraduate 
education and if it should be institutionalized in the DOD. 

3. Propose a model for T&E postgraduate education for 


military and civilian personnel. 


CC. AMEOTHESLS 
The T&E of complex weapon systems and sub-systems requires 
unigue skills to understand, design, conduct, evaluate, and 


report test results. The T&E of weapon systems in support of 





the DOD acquisition cycle has evolved into a recognizable 
engineering discipline. The professional technical personnel 
in the DOD T&E community should be considered as unique 
assets and should be supported by establishing a postgraduate 


curriculum in T&E engineering. 


De SCOPE 

This research considered the evolution of T&E in the DOD 
in the 1970's to the beginning of the 1980's. The T&E of air- 
craft weapon systems in the Department of the Navy (DON) was 
used as the primary means for illustrating the objectives of 
this research. If a contiuum of need (from minimum to maximum) 
for T&E postgraduate education was applicable, the T&E of 
aircraft weapon systems would fall in the maximum need posi- 
tion of this scale. This constraint does not detract from the 
universality of the results to the role of T&E in support of 
other types of weapon systems acquisition in the DOD. Although 
each of the Services does not face exactly the same conditions 
in acquisitions of weapons systems, the tasks and problems of 
T&E are similar. Departments of the Army and the Air Force 


T&E perspectives were used when appropriate to this study. 


E. METHODOLOGY 

The approach used to prepare this study was threefold. 
The author relied heavily on 15 years of experience in the 
T&E of aircraft weapon systems. An archival search was con- 


ducted for references relative to this study. Interviews 
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were held with Naval Postgraduate School faculty, high level 


DOD officials, and T&E field activity personnel. 


F. REVIEW OF RELATED LITERATURE 

Two previous Naval Postgraduate School (NPS) theses [Refs. 
1,2] deal with the T&E of Navy aircraft weapon systems and 
are related to this study topic. 

In reference 1, LCDR Bowes evaluated the Navy T&E process 
for aircraft weapon systems, identified deficiencies and good 
features of this process, and proposed a reorganization of the 
Navy's T&E organizational structure. One of his conclusions 
was the need for a course of instruction in the discipline of 
weapon system T&E. In addition he recommended that a Systems 
T&E Training School should be established at each of the major 
test centers resulting from reorganization of the Navy's T&E 
activities. This thesis presents the first documented need 
for formal Navy T&E education. 

In reference 2, LCDR Stoll examined the tasks assigned to, 
the level of knowledge and experience of, and the need for 
formal education of project officers at aviation T&E activities. 
He proposed a T&E education syllabus and alternatives for 
achieving it based on the results of a survey of 78 project 
officers located at development and operational aviation T&E 
activities. Seventy-three officers were then attached to the 
test activities and five were attending NPS but had previous 


T&E experience. 
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He reviewed test reports from the test activities to 
determine the areas of knowledge and compared these with areas 
of knowledge obtained from interviews and questionnaires. 
Three out of four officers had received inadequate or no formal 
training in the pertinent subject areas. One out of two 
project officers, in their first year of experience, indicated 
that their lack of T&E knowledge reduced their effectiveness 
to do their jobs. As a result of the interviews and question- 
naires, the pertinent subject areas were reduced from 61 to 
49. These results substantiated the need for formal T&E 
education. 

LCDR Stoll designed a syllabus consisting of 10 courses 
and 110 instructor contact hours. Six alternatives were pro- 
posed. The alternatives range from a four to six quarter in- 
residence T&E curriculum at the NPS (least favored by the 
project officers, since they would rather be flying) to local 
programs conducted by the test activities. He concluded that 
One approach would not satisfy the need for formal T&E educa- 
tion, consequently, a combination of methods was the recommend- 
ed approach. An idealized approach was presented which con- 
sidered eventual implementation of all the proposed alterna- 
tives as a solution to this dilemma. 

Since these theses were completed, the Naval Air Test 
Center was reorganized and the Naval Air Systems Command 
established an AIR-06 T&E Coordinator Division. It's mission 


is to coordinate and manage the Naval Air Systems Command 
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TSE resources which include the field activities and the 
major test ranges. In addition, the T&E Coordinator integrates 
the operational T&E command test requirements into the Test 
and Evaluation Master Plan for aircraft weapon systems. The 
theses do not address the education requirements of the 
Civilian T&E population which is a major portion of the T&E 
community. 

Reference 3 is related in several ways to this study. 
Dr. Matthews provides a compilation of T&E related material 
which he prepared to focus attention on the evolving nature 
of T&E. He advocates the recognition of T&E as an engineering 
discipline. From his viewpoint T&E in the DOD is an extension 
of basic scientific and engineering disciplines. One of his 
observations was that there is a need for formal T&E education 
and he proposed several alternatives for implementing it. The 
emphasis of this document is toward providing a reference 
source for background information on T&E in the DOD and an 


aid to promoting T&E as an engineering discipline. 


G. DEFINITIONS 

Each of the Services over the years has introduced ter- 
Minology pertaining specifically to its efforts in T&E. The 
Joint Logistics Commanders (consists of Commander U. S. Army 
Materiel Development and Readiness Command, Commander Air 
Force Logistics Command, Commander Air Force Systems Command, 


and Chief of Naval Material Command) was formed to promote 
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cooperation in joint service acquisition programs. An ad 
hoc group on T&E planning guidance published the Compendium 
of Test Terminology [Ref. 4]. This document provides a use- 
ful working level reference to avoid confusion when the same 
term is used by each Service. The following definitions 
define the three types of T&E most frequently referred to and 
defined in reference 5. 

imereveropment Test and Evaluation (DT&E) 

DT&E is that T&E conducted to assist the engineering 
design and development process and to verify attainment of 
technical performance specifications and objectives. DT&E is 
usually accomplished or managed by the DOD Component's material 
agency. It includes T&E components, subsystems, hardware/ 
software integration, related software, and prototype or full- 
scale engineering development models of the system. T&E of 
compatibility and interoperability with existing or planned 
equipment and systems are also included. 

Pee peraclonal test and Evaluation (OT&E) 

OT&E is that T&E conducted to estimate a system's 
Operational effectiveness and operational suitability, identify 
needed modifications, and provide information on tactics, 
doctrine, organization, and personnel requirements. Acquisition 
programs shall be structured so that OT&E begins as early as 
possible in the development cycle. Initial Operational Test 
and Evaluation (IOT&E) must be accomplished prior to the 


Milestone III decision (production). 
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See roaduction hecevranece Test and Evaluation (PAT&E) 
PAT&E is T&E of production items to demonstrate that 


procured items fulfill the requirements and specifications of 
the procuring contract or agreements. Each DOD Component is 


responsible for accomplishing PAT&E. 
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It. BACKGROUND 


A. GENERAL 
The weapon system acquisition policies of the DOD in the 

late 1960's received a lot of public attention. DOD had earned 
a reputation for cost growth, schedule delays, and production 
of systems before they were ready. This increased visibility 
of DOD's perceived mismanagement of the defense tax dollar 
resulted ina study of the entire organization, structure and 
operation of DOD [Ref. 6]. A part of this,study was a review 
of the role of Test and Evaluation (T&E) iny the acquisition of 
weapon systems. A conclusion of the study was: 

"Operational test and evaluation has been too infrequent, 


poorly designed and executed, and generally inadequate." 
[Ref. 6:2] - 





To focus additional attention on test and evaluation the study 
recommended a Defense Test Agency which would oversee all 
Defense T&E. Specifically it would design or review the 
designs for tests; monitor the entire Defense test program; 
and conduct T&E of systems which can be used across Services. 
Also, the Panel recommended that a separate funding category 
Should be identified for T&E and operational testing. The 
Defense Test Agency would have responsibility for this 
funding. 

Deputy Secretary of Defense, David Packard, was instru- 


mental in implementing recommendations of the Blue Ribbon 
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Defense Panel which have affected T&E. He established the 
Director of Defense T&E and moved the Services to establish 
independent test agencies which report to the Service Chiefs 
[Ref. 7]. Additionally, he increased the role of OT&E results 
in the acquisition process by issuing DOD Directive 5000.1 
[Ref. 8] requiring that OT&E be conducted before a decision 

1s made on full scale production. Congress also gave momentum 
to the changing role of OT&E by passing a law which requires 
that DOD give Congress, annually, the results of OT&E with 

its request for funds on systems it is planning to procure 
[Ref. 9]. The DOD Directive 5000.3, Test and Evaluation 

(Ref. 7], was issued to define policy on the types of testing 
and the roles of the Services in T&E. This directive also 
established the requirement for a Test and Evaluation Master 
Plan (TEMP) which coordinates all the testing requirements 

for a given program. 

The Defense Science Board Task Force on T&E was formed in 
late 1972. It was tasked to study a group of major weapon 
systems to determine whether improvement in T&E would permit 
earlier discovery of problems and initiation of corrective 
feetemomeits report [Ref. 10] in early 1974 identified that 
improvements were needed in test planning and scheduling. T&E 
participation was frequently deleted due to overruns of time 
and money in other phases of the program. The Panel described 
a T&E gap which occurred when testing pre-production hardware 


Or prototypes. Since the decision to start production cannot 
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occur until after initial OT&E (IOT&E) is completed, a 
substantial lag (1.5 to 2 years) exists between completion 
of development testing and manufacture of the first produc- 
tion article. The Panel also provided nine related reports 
which provide T&E guidelines for different types of systems, 
for example, for missile weapon systems and for aircraft 
systems. 

The Office of Management and Budget published OMB Circular 
A-109 in 1976 which outlined the government's policy on the 
acquisition of major systems [Ref. 11]. This circular was 
1ssued to curb cost overruns and ensure the new systems are 
needed. As a result of the issuance of this circular the DOD 
and Service directives and instructions had to be revised. No 
major changes in T&E policy were generated except for the 
reemphasis of the requirement for T&E criteria. This included 
the concept of independent testing and testing in the opera- 
tional environment prior to approval for full production. 

The lengthening of the acquisition cycle was studied by 
a Task Force of the Defense Science Board and reported in 1978 
[Ref. 12}. It concluded that sequential and separate testing 
of the systems by the test activities of the developer and the 
independent test activities representing the user often adds 
unnecessary costs and time. Thus the Task Force recommended 
conducting combined development and operational tests wherever 
feasible and separate evaluations of the data. The Task Force 


expressed concern that independent test activities with their 
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increased influence would demand duplicate facilities and 
increased testing. What is really desireable is combined 
within the sane service) testing and independent evaluation. 
Also, it was felt that the T&E activities should participate 


throughout the entire acquisition cycle. 


B. ENVIRONMENT 

Presently, two major Navy aircraft weapon systems are in 
full scale development testing at the Naval Air Test Center, 
Patuxent River, Maryland. The F-18 program is employing the 
Single-site test concept. The contractor performs its develop- 
ment and operational testing at this government facility with 
Navy test activities monitoring the contractor's tests. At 
specified times each of the Navy's test activities conduct 
testing and perform independent evaluations of the data. 

This approach has numerous advantages and disadvantages which 
have not been fully assessed. 

The second major aircraft weapon system in full scale 
development testing at the Naval Air Test Center is the Light 
Airborne Multi-Purpose System (LAMPS) MK-III. The LAMPS 
MK-III helicopter and airborne systems are an extension of the 
ship's Anti-Submarine Warfare (ASW) capability. In the case 
of the LAMPS MK-III the Naval Air Test Center is not a single 
Site test location. It functions as the focal point for all 
the Navy field activities' participation in the testing of 


the complete LAMPS MK-III weapon system (includes aircraft and 





ship systems). The responsible operational test activity is 
participating in combined testing with the developer's test 
activities to reduce duplication, time, and costs of testing. 
The operational test activity still retains designated test 
periods for Initial OT&E (IOT&E) and Follow-on OT&E (FOTS&E). 
The Test and Evaluation Master Plan (TEMP) was utilized as a 
major management tool in working out the participation of the 
T&E activities in this program. The flow f instructions 
which govern the preparation of the TEMP is shown in Appendix 
mee Figure l. 

The T&E participation in the minor acquisition programs 
normally does not fare very well. It is estimated that 
Minor acquisition programs amount to three times the major 
programs. The management, however, of these programs 1s not 
normally provided with sufficient resources or experience to 
manage the acquisition until it runs into trouble. The 
Services have initiated the same acquisition procedures for 
Minor programs as for major programs with lower level review 
and approval requirements. Consequently, the preparation of 
a TEMP to coordinate the T&E effort is an exception rather 
than a rule. Few, if any program management offices, have 
personnel experienced in T&E. The TEMP is a useful test 
coordinating document, because, it brings the players together 
to negotiate their requirements. It must be remembered, 
however, that the acquisition cycle is dynamic, and the TEMP 


1S a point-in-time document. The program office must therefore 
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become innovative in its effort to communicate changes in 


the TEMP to all the participating activities. 


ee LECHNOLOGY 

The major and minor acquisition programs normally incor- 
porate the latest technological advances. American industry 
is the world leader in developing new technology and in 
bringing it to the market place. High technology is driving 
the economy while all segments of industry scramble to apply 
it to increase productivity and drive down production costs. 
This same pyschology exists in the acquisition of weapon 
systems. 

The T&E community personnel are dealing with increasingly 
sophisticated and complex equipment and systems. The use of 
airborne computers and associated software have found applica- 
tion in on-board processing of information and analysis of 
acoustical data in ASW missions, navigation, weapons delivery 
control, communication, etc.. Rapid advances in semi-conductor 
devices continues at a rapid rate. Functions which previously 
could not be accomplished because of the size of the resulting 
hardware are now being incorporated into airborne platforms. 
Flight-control systems are now built around airborne computers 
vice mechanical linkages. How the aircraft flies is a function 
Seeenewapolication of complex mathmatical control theory and 


programming skills. 
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Airborne Command, Control, Communication and Intelligence 
(c7 I) functions are performed airborne as well as on the 
ground. Achievement in this area has leaped forward with the 
incorporation of semi-conductor advances into airborne radar 
and communications devices. Likewise the electronic warfare 
capabilities to detect, jam,or fool the agressor have all been 
enhanced by the incorporation of semi-conductor technology. 

Material technological advances in ceramics, composite 
and metal compounds are being used in airframes to increase 
strength to-weight-ratios, reduce detectability, and to 
increase reliability and maintainability. Technological 
advances in ceramics and metal compounds, producing higher 
temperature resistant materials, are being applied to propul- 
Sive devices. As a result, jet engines are producing higher 
thrust-to-weight ratios and are more energy efficient. 

The technology list that the T&E community must deal with 
is lengthy. The T&E of these high-technology systems presents 
unique testing challenges. The challenge facing the T&E 
community is to stay technically competent to do effective 
testing in the face of fixed personnel ceilings and increasing 


workload. 


D. ORGANIZATION STRUCTURE 
The Secretary of Defense (SOD) established the Director 
of Defense for Test and Evaluation (DDT&E). The DDT&E was 


designated to report through the Under Secretary of Defense 
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for Research and Engineering. The DDT&E was identified as 
having responsibility for monitoring and advising the SOD on 
T&E conducted by the Services. It also provides access to 
the office of the SOD for the Services' OT&E activities on 
Matters concerning independent testing. It is unlikely that 
the military OT&E activities would use this access without 
going through their normal command structures first. The 
DDT&E organizationally is also the focal point for joint 
service testing programs. 

The organizational changes which resulted from the increased 
emphasis on OT&E had the least impact on the Navy. The Navy 
already had the Operational Test and Evaluation Force (OPTEVFOR) 
in being. Therefore, it was able to comply rapidly to Deputy 
Secretary of Defense Packard's request for the Services to set- 
up independent operational testing agencies. The Navy establish- 
ed a Director of RDT&E, OP-98, with the Commander of OPTEVFOR, 
OP-983, reporting to the Navy Director of RDT&E. This 
Organizational change was the Navy's compliance for the 
operational test activity to report to the Service Chief 
independently of the developing agency. 

The Naval Air Systems Command organizational structure was 
amended in 1975. An Assistant Commander for Test and Evaluation 
Coordination was established [Ref. 13]. This organization had 
existed previously at a lower organizational level and was 
tasked with allocating the Navy's RDT&E aircraft resources. 


This organizational change was partially attributed to the 
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findings of the Donaldson Committee Report on the T&E base 
study, conducted for the Commander Naval Air Systems Command 
[Ref. 14]. A similar type of organization for T&E coordina- 
tion is not used by the other Services. 

The Army and the Air Force required several years, until 
the mid 1970's to comply. The Army established the Operational 
Test and Evaluation Agency (OTEA) and the Air Force set up the 
Air Force Test and Evaluation Center (AFTEC). The Commander 
of OTEA reports to the Army Chief of Staff and the Commander 
of AFTEC reports to the Chief of Staff Air Force. Both OTEA 
and AFTEC have small headquarters' staffs which are dependent 
on the operating commands to provide the manpower resources 
to support OT&E. The Army and Air Force started out emphasiz- 
ing combined testing in order to reduce costs and to use pre- 


production resources more efficiently. 


fee PLAYERS 

1. Program Manager 

The program manager is the hub of the acquisition 

cycle and the key to the level of T&E participation. He must 
budget and plan for this participation and provide for any 
special resources such as facilities which could involve 
military construction funding. Normally the program manager 
changes many times during major weapon system acquisitions. 
Since it takes from 12 to 15 years from concept initiation to 
initial operational capability, it is not unusual for a program 


to have five program managers. 
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The program managers seldom have previous T&E experience 
and it is easy for an adversary relationship to develop between 
the program manager and the test community. He frequently does 
not understand the functions of the Major Ranges or the Test 
Centers and how to use them to support the development effort 
or reduce engineering risk. The test activities by reporting 
what iS wrong with the system increase the program office 
workload to explain away the problems or to plan for funding 
and schedule changes to correct the deficiencies. 

The military specifications and policy instructions 
which are part of most weapon system contracts have built-in 
test requirements which often come as a surprise to the 
inexperienced program manager. Viewing T&E from the program 
manager's perspective, his main objective is to get the OT&E 
activity to grant a Provisional Acceptance for Service Use 
(PASU) or an Acceptance for Service (ASU) which is the more 
desireable. In the real world of politics, funding availability, 
and schedule crunches the tendency is to make decisions favor- 
ing OT&E requirements at the expense of DT&E requirements. 

Program managers for minor programs are normally even 
less prepared than the managers of major programs. Even 
though the minor programs account for approximately three times 
the funding of major acquisition programs, the program manager 
is frequently a technical type who has not developed the 
Management and technical competencies to conduct a minor 


acquisition. He must write the technical requirement portion 
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of the Request for Proposal, evaluate proposals, write the 
work statement, provide technical direction to the development 
effort, write test requirements, plan the conduct of the 
engineering development tests, and provide guidance to higher 
management. The minor program manager historically is compet- 
ing with the major programs for funds inadeguate to support 
the procurement of all the items in deveopment. Testing 
appears aS an expensive item ina limited funding situation 
or in an inadequately planned program and is usually among the 
first items to be compromised. Minor programs, due to lower 
visibility, are even more susceptible to frequent program 
Manager turnover. One of the findings of a GAO report con- 
ducted on the Defense System Acquisition Review Council (DSARC) 
was a relatively rapid turnover of high ranking DOD and Service 
personnel responsible for the integrity of the DSARC process 
ieecewels|). High ranking DOD civilians and military personnel 
were changed at an average of 28 months and 24 months respec- 
tively. This data certainly reflects unfavorably on the 
management of the acquisition cycle and is relatively representa- 
tive of program manager turnover. Unless the T&E program is 
documented in a TEMP in the early phases of the acquisition 
cycle, T&E is always vulnerable to inadequate funding and 
support. 
Zeoaeeetsces Testers 
The contractor for the weapon system conducts testing 


during the acquisition to support engineering development 


26 








efforts. The government personnel employed by the Major 
Ranges and Test Facility Base (MRTFB) are primarily respon- 
Sible for conducting the government's technical testing during 
the full scale development and production phases of the 
acquisition cycle (Appendix A, Figure 2). The developer's 
test activities also participate in the earlier phases of the 
acquisition cycle but at a lesser degree than laboratory or 
development center personnel. A listing of MRTFB and other 
activities involved in T&E are listed in Appendix A. 

It is difficult to determine the exact number of 
government personnel employed in T&E. The civilian ceilings 
have declined some in recent years, however, with an increas- 
ing work load the test activities have turned to contracting 
to operate test ranges and for some engineering support. 
Collectively the DOD RDT&E community totals in excess of 
100,000 personnel [Ref. 16]. The DDT&E estimates that approx- 
teeecisa 50,000 military, civilian, and contracted for personnel 
operate the MRTFB and provide special test support. These 
figures do not include OT&E personnel such as those assigned 
to COMOPTEVFOR and the operating forces assigned temporarily 
to the Army and Air Force operational test activities. 

The MRTFB is staffed with both military and civilian 
engineers and scientists. It is estimated that they will 
Spendec tele oiliien in FY 1981 [Ref. 17] not including the use 
of Operating and Maintenance (O&M) funds -- O&M funds support 


the weapon system once it is in production to the end of its 
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life cycle -- to conduct T&E of weapon systems. In terms of 
government personnel and funds T&E is big business. 

The DT&E community in the past has had difficulty in 
obtaining engineering and scientific personnel due to a 
combination of factors. A declining number of graduates in 
engineering and the sciences, remote locations of the DT&E 
activities, reduced personnel ceilings, hiring freezes which 
frustrate recruiting efforts, and starting salaries substant- 
lally below industry entry levels have summed to weaken the 
DT&E technical base. Military personnel, specifically Navy, 
with previous T&E experience are less likely to return for a 
second tour of the DT&E activities due to a lack of a defined 
career path in T&E. This is especially true in the DT&E of 
Navy aircraft weapon systems. Costly and extensive training 
is provided to naval flight officers to develop engineering 
test pilot skills. In turn these officers serve an approximately 
two-year tour before moving on to an assignment unrelated to 
their T&E skills. 

DTSE skills are mostly learned by on-the-job training 
(OJT). Each of the DT&E activities have established the 
methodologies and techniques, instrumentation, and facilities 
to acquire hard data to make assessments of engineering develop- 
ment and risk. Most of the DT&E activities are in remote 
areas and are limited markets for advanced technical education. 


This has constrained the access of the DT&E personnel to 
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advanced technical education needed to renew their engineering 
skills to challenge the complexity of modern weapon systems. 

The Navy DT&E of aircraft weapon systems uses skilled 
Military operator personnel. Even though DT&E is conducted 
under more controlled conditions than operational testing, 
military test personnel are qualified to relate test results 
to expected performance under fleet operating conditions. 
The DT&E results are reported to the development agency and 
OT&E personnel are encouraged to use DT&E results where feasible. 
Also, DT&E personnel are required to support OT&E. In contrast 
OT&E results are reported to the Chief of Naval Operations 
level and are not normally available to DT&E personnel. The 
reporting of weapon system deficiencies by DT&E personnel are 
not necessarily reported in the same form as by OT&E personnel. 

3. OT&E Testers 

T&E is a team effort consisting of the program manager, 
the development test activities and the operational test 
agencies. Most of the focus, however, in the 1970's has been 
Oman increase of the role of operational testing. The 
acquisition policy has been changed to include participation 
of OT&E personnel throughout the acquisition cycle. The 
Services have used different approaches to meet the increased 
responsibilities of OT&E. The Navy has its operational test 
agency headquarters in Norfolk, Virginia. It has staffs on 
the East and West Coast, three operational aircraft squadrons 


(based on type of aircraft and mission), and detachments -- 
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as required to support new tactics, operating procedures, and 
techniques. The Army's operational test agency is located at 
Falls Church, Virginia and the Air Force operational test 
center is located at Kirtland Air Force Base, New Mexico. The 
Air Force and Army staffs are small in comparison with the 

Navy (less than 100 personnel vice more than 1400 for the 

Navy). None of the Service OT&E agencies employ many civilians, 
therefore, continuity in the lengthly acquisition cycle is 
difficult to maintain unless military reassignments are made 
less frequently. The Navy does not provide for extended tours 
for military personnel assigned to OPTEVFOR. [In fact this 
would be contrary to the policy stated by COMOPTEVFOR [Ref. 18]. 
This memo expresses the importance of having skilled operators 
with recent "blue water" experience vice highly trained 
technical operators to conduct operational testing. The Navy 
OT&E concept was used by the Blue Ribbon Panel as an example 
for the other Services to consider. The Navy OT&E was faulted 
for its inability to produce "hard numbers" to assist high 
level decision making. The COMOPTEVFOR has employed a few 
Siviiivansanalysts to improve on this situation. 

The GAO in 1977 began a review of the Service OT&E 
efforts. The Navy, having the most experience with an OTS&E 
organization, was reviewed first [Ref. 19]. GAO found the 
Navy was still making production decisions with incomplete 
OTSE results. OT&E could not be conducted on some systems due 


to a lack of test resources, and OT&E conducted on systems, 
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after production approval was granted, found performance 
problems that should have been corrected before production 
approval. 

OT&E agencies feel they are the only test activities 
which test the complete weapon system. OPTEVFOR views are 
expressed on this position in the Operational Test Director 
Guide [{Ref. 20]. This manual also raises an interesting point 
on the OPTEVFOR philosophy on the availability of operational 
test data: 

"In some programs it may be convenient (or absolutely 
necessary) to use DA field agencies to get OT&E data 
reduced and, to some degree, analyzed. In these situa- 
tions it 1S mandatory that these people be under the 
Seebatrenal control of COMOPTEVFOR (represented by the 
OTC/OTD/Program Analyst) while they are working on OT&E 
data. Their work is defined in advance, and their results 
are furnished only to COMOPTEVFOR, unless COMOPTEVFOR 
has specifically approved a wider distribution." [Ref. 20: 
16-2) 
Field agencies are required to provide all information to 
OT&E personnel and it could be implied from the above that the 
exchange of data is in one direction only. 

Although COMOPTEVFOR's earlier position was to shun 
the use of operator personnel with technical backgrounds, the 
extreme pressures to make production decisions using “hard 
data" has weakened this position. In the OT&E of aircraft 
weapon systems there is trend toward OPTEVFOR project personnel 
having technical backgrounds. This has the potential to 
change the OT&E viewpoint toward acquiring more hard data to 


substantiate OTSE results; and it raises issues on duplication 


between OTSE and the developing agency's test activities. 
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The Air Force and the Army have approached OT&E 
differently than the Navy. They have emphasized combined 
test activities of the developmental and operational testers, 
the sharing of limited pre-production resources, and the use 
of highly skilled operational testers. The GAO has critisized 
Paedr POlicies on OT&E [Refs. 21,22]. In particular, both 
Services needed to conduct operational testing in environments 
more representative of the operational environment. Also, the 
Services should designate distinct periods for operational 
testing and provide the necessary test resources. The test 
personnel and support should be more representative of the 
type used when the system is deployed. 

4. Board of Inspection and Survey and AIR-06 

Two agencies which are unique to the Navy are the 
Board of Inspection and Survey (BIS) and the Naval Air Systems 
Command (AIR-06), the Assistant Commander for Test and Evalua- 
tion. The President of BIS is located in Washington, D. C. 
and has Sub-Boards of Inspection and Survey located in Virginia, 
California, and Maryland. The Sub-BIS for aircraft is located 
at Patuxent River, Maryland. It tasks, for example, the Naval 
Air Test Center to conduct technical tests of production air- 
craft weapon systems or follow on tests on significant changes 
to aircraft in use. Based on the test results the President 
of BIS recommends to the Secretary of the Navy whether the 
aircraft weapon system should be accepted for fleet use. 


The potential conflict or duplication of the BIS and 
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COMOPTEVFOR roles have been actively discussed and analyzed 
recently. The Navy expressed confirmation of the need for 
each organizations' function, role and contribution to the 
T&E process through the recent revision of OPNAVINST 3960.10A 
fers 23). 

The AIR-06 organization evolved from the need to 
provide a focal point for NAVAIRSYSCOM RDT&E resources. AIR-06 
coordinates the Navy's Major Range and Test Facility Base 
(MRFTB) with DDT&E. Since the TEMP has become a key document 
for major weapon systems, AIR-06 has taken responsibility for 
coordinating the DT&E and OT&E requirements for the program 
Manager. AIR-06 also coordinates the workload going to the 
NAVAIR field activities with the resources required to conduct 
DTSE. The effectiveness of this organization in performing 
its mission has not been confirmed since its establishment in 


OND. 


Eee EVOLUTION 

There is presently a significant variance between the Navy 
Beppeoach to TSE and those of the Air Force and Army. In the 
1970's the decision making processes for major acquisition of 
weapon systems has varied from decentralization to the Services 
and then a return to centralization at the Office of the 
Secretary of Defense. Presently, the trend is toward account-— 
ability and decentralization to the Services as outlined in a 


recent Deputy Secretary of Defense memo [Ref. 24]. Consequently, 
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high visibility issues, such as, productivity improvement by 
increasing DOD effectiveness and efficiency through reduction 

in duplication of Service efforts will require more cooperation 
between the Services and more joint Service acquisition programs. 
This will magnify the differences in the Services' approaches 

to T&E and result in a buildup of forces toward more continuity 
in test techniques, methodologies, test personnel qualifications, 
and analytical techniques. The pressure by DOD for hard 
Operational test data will increase. This will force the Navy 
to make substantial compromises in its past policies on indep- 
endent operational testing and its need for more analysis in 

the evaluation phase. The Services will approach a common 

T&E base to allow DOD to make rapid management decisions to 
reduce the length of the acquisition cycle. DDT&E's future 

role is unclear in the present move toward decentralization. 
Since at present DDT&E operates primarily in a persuasive 

mode with the Services, increased joint Service programs 

might necessitate more authority for the DDT&E. A Defense 


Test Agency could even result [Refs. 6,16]. 
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III. GRADUATE EDUCATION 


A. GENERAL 

miesneed for full-time, fully-funded graduate education 
for DOD personnel has received considerable attention in the 
1970's. The declining enrollments in higher education has 
accelerated the availability of graduate education at DOD 
facilities as colleges and universities have competed for 
this source of revenues. Advances in technology and 
society's acceptance of the value of education has increased 
this need for continuing education by the work force. Most 
who seek advanced education do so to receive a graduate degree. 
Military positions have been established requiring advanced 
degrees to perform the duties of the job. This frequently 
has been done to enhance retention, increase the morale of 
the personnel and provide for increased job opportunities 
after retirement. On the civilian side, the bureacracy has 
used advanced degrees as a screening mechanism in selection 
for promotion and to increase the technical status of the 
organization. 

Before most personnel make a commitment to long term 
training one of their chief concerns is whether the award 
of a graduate degree can be achieved. Since advanced degrees 
are becoming a norm in our society, it is unlikely there will 


be many participants in a full-time advanced education program 
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just for the benefit of the additional training. The selec- 
tion of civilians for long-term fully-funded training, as a 
recognition for having high-potential to be future hign-level 
leaders within DOD, places additional pressure on the recipient 
to earn an advanced degree. The sponsoring activities also 
subtly encourage that the advanced degree is an expected 
mesmee OF the trainee's efforts. The T&E activities, primarily 
those involved in development testing and evaluation, have 

long recognized and voiced the need for graduate education in 
Support of the T&E process. The engineering schools have 

been reluctant to offer off-campus programs which lead to 
advanced degrees mainly for traditional and quality of gradu- 
ate education reasons. Most T&E activities are in remote 
locations and provide a small market. This does not motivate 
the engineering schools to be flexible in their attitudes 
toward serving the needs of the T&E community. An MIT study 
found there was little research available to evaluate the 
performance of engineering schools in meeting the education 
needs of their graduates [Ref. 25]. Most engineering schools 
have been too preoccuppied with spending Federal research 
dollars or graduating students to meet industry's requirements. 
They have consequently failed to solicit graduate feedback. 
The Accrediting Board for Engineering and Technology, in 
order to prevent stagnation in engineering education, does 

not advocate rigid standards for accreditation and encourages 


well-planned curriculum experimentation [Ref. 26]. 
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The literature reviews cited in the Introduction proposed 
programs for graduate education in T&E. The Pacific Missile 
Test Center, a Navy T&E Piyedlel activity, located at Point 
Mugu, California has also proposed some innovative programs 
to provide graduate education to its technical employees. 
The Naval Postgraduate School (NPS) in Monterey, California 
has two courses listed in its catalog which are titled Test 
and Evaluation which are taught by the Operations Research 
Department. In the past, informal discussions with Navy 
field activities’ personnel have resulted in NPS spokesmen 
proposing "strawmen" T&E curricula. These proposals have 
not moved forward due to lack of Department of the Navy 
Support and current constraints on NPS resources. The Naval 
Aviation Executive Institute Program was established at Naval 
Missile Test Center, Point Mugu, California in 1975 and 
moved to the NPS in 1976. This program has high potential 
to respond to the graduate education needs of the Navy's 
T&E community and could provide a model for the rest of the 
T&E community. 

Some observations are presented on the status of T&E 
engineering as a profession. The parameters which have 
been identified that pertain to T&E of aircraft weapon 
systems but are relevant to the T&E of other weapon systems 
and equipment are discussed. The needs of different levels 
of personnel in the Navy T&E community are reviewed and 
a framework presented which could support T&E educational 


needs. 
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See) FULL-TIME, FULLY-FUNDED 

The atmosphere for full-time, fully-funded graduate 
education in the DOD has not been very favorable. The NPS 
which in the eary 1970's had an enrollment of 1200 Navy 
students has had its budget cut by Congress to where the NPS 
had 400 students in FY 1977. This figure has increased to 
600 Navy students for FY 81 and the Congressional committees 
are in support of this level. The Navy and Air Force have 
been critized for the cost per student to provide postgraduate 
education for their personnel compared with the cost per 
student of using non-military institutions. The Navy has 
been the smallest user of public sector colleges and universi- 
ties to provide full-time, fully-funded education. The Air 
pomee, although it supports the Air Force Institute of 
Technology (AFIT), sends a majority of its full time graduate 
students to the public sector institutions. The Army sends 
most of its full time students to the public sector institu- 
tions except for those it sends to NPS and AFIT. 

The GAO [Ref. 27] addressed the permissive attitude of 
the services toward full-time, fully-funded graduate educa- 
tion for officers. The services were tasked to validate 
positions requiring advanced education in order to perform 
the duties of the position. These were primarily positions 
in the science and technology areas. The intent of providing 
the postgraduate education was to prepare officers to function 


more effectively in these positions. The GAO report took 
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issue with the long education periods, some extending to 3 
years. Also the GAO noted that equivalent positions in the 
Services were judged inconsistently regarding the need for 
graduate education. Some positions required advanced educa- 
tion and others did not. Positions were validated as requir- 
ing advanced education only because the officer in the 
position had an advanced degree. Official job descriptions 
requiring advanced degrees were submitted for validation 
which did not address the requirement for graduate education 
in the job descriptions. The GAO found from its sample audit 
of military installations that officers with graduate educat- 
10n were not being assigned to the positions which required 
graduate education. 

The GAO recommended that the Services use short courses 
and work experience as alternatives to full-time graduate 
education. Also they should fill the positions with civil- 
ians where possible. DOD was requested to issue a new policy 
restricting the number of positions requiring a graduate 
education. Congress was requested to control the number of 
feerearyeorlicers in full-time, fully-funded graduate study 
by limiting the funds available. 

DOD had encouraged the graduate education of military 
officers. In our society there are generally accepted values 
and benefits of graduate education. DOD felt that the 
graduate education enhanced the effectiveness and capability 
of the officers and was an important factor in retaining 


them. 
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The GAO considered the effectiveness of long-term, 
full-time graduate education for DOD civilians in non- 
government institutions [Ref. 28]. Long term training was 
considered training greater than 120 days. The training 
was through courses at the graduate level primarily in 
management, scientific, engineering, or technical fields. 

The DOD installations were critized because they did 
not follow existing regulations. The field activities had 
not determined what their near and long term (5 year) train- 
ing needs were nor had they prioritized them. The GAO 
found little evidence that there was a plan to use the new 
Skills of the employee upon return from long-term training. 
In addition, the field activities had: a) not given much 
thought to selecting employees for training in areas where 
there was the greatest need; (b) overemphasized the require- 
ment to obtain an academic degree; (c) not used any criteria 
for selecting non-government schools; and (d) not provided 
for any evaluation of the training to determine if its 
objectives were met. 

The objectives of DOD's support of long-term, full-time 
training and education for civilian employees are as follows: 
(a) to maintain a position of leadership in defense-oriented 
science and technology; (b) to provide opportunities for 
career employees of promise to grow and realize their full 
potential; and (c) to better enable employees to success-— 
fully cope with the complex problems of managing all aspects 


of national defense activities. 
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The Civil Service Commission, DOD, and the Service's 
regulations emphasize that training qualify personnel for 
performance of official duties. The accomplishment of an 
advanced degree should be an incidental by-product of the 
training. The GAO found, however, the field activities' 
applications for long-term training implied that the indivi- 
dual was expected to achieve an advanced degree. Training 
programs were frequently structured so that they lead to an 
advanced degree, hence improving the prestige and image of 
mie activity. Additionally, GAO found that the individuals 
applying for long-term training did so out of self-interest 
and their self-development did not support the training 
objective of the activity. The individuals selected their 
Own course of study and selected the training institution. 

The National Academy of Public Administration conducted a 
Study on military officer graduate education in 1975 [Ref. 
29}. This was just prior to congressional action on an OMB 
recommended 70 per cent reduction in funding for full-time, 
fully-funded graduate education. The panel concurred that 
the military should have educational background comparable 
with their counterparts in the private sector and government. 

The existing system of counting positions requiring gradu- 
ate education, however, should be replaced with one that sets 
educational standards and integrates the need with technical, 
organizational, and strategic plans. It recommended the 


Services utilize other programs such as off-duty, recruitment 
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of officers with advanced degrees -- especially in fields with 
Shortages -- and cooperative study (flexibility for officers 
to take courses during normal working hours). The Navy and 
Air Force postgraduate schools were encouraged to increase 
their number of off-campus personal study courses so as to 
reduce the time in residency to complete graduate programs. 
In order to accomplish more off-campus education with high 
standards and quality, representatives of the Services and 
higher education should establish ground rules for instructor 
selection and credit transfer. The report emphasized the 
feee £0r Classroom situations with both civilian and military 
participation in the learning process. 

The report noted that the need for graduate education is 
obvious in the physical, mathmatical, and engineering sciences. 
These disciplines are used in the development and operation 
of high technology weapons, transportation, and communication 
systems. In addition, advanced education is necessary in 
biological, social, operational, and managerial services in 
order to manage, deploy, and operate these systems. Although 
the Services depend on civilian industry to provide the 
talent to produce the weapons they need, an officer corp less 
educated than its counterparts will be severly handicapped in 
meseselattons with civilians. 

In a Situation of reduced funding, technical advanced 
education is the beneficiary of a majority of the funds for 


full-time, fully-funded education. It is nearly impossible 
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for an officer to acquire an advanced degree in engineering 
or science through an off-duty or a partially-funded program; 
and it is easier to couple specific technical and science 


Gegrees with validated billets. 


C. AFTER WORKING HOURS 

mae proliferation of after hours graduate programs has 
raised questions about the quality of these programs. Most 
installations have taken the view that having some graduate 
program offerings is better than nothing. These programs 
stretch out for several years and have high student drop- 
out rates. A majority of the programs are management vice 
technical oriented because most engineering schools have 
been reluctant to offer complete off-campus graduate 
programs. 

There are some limitations to graduate education conducted 
under these conditions. Most employees/students are at a low 
point in their learning capacity and retention capability 
after a day's work. The employee's class attendance is 
frequently interrupted by work-related travel or problems. 
The classes consist primarily of work associates; therefore, 
there is not the opportunity to experience the multiple 
Viewpoints occurring in the campus environment. Specific 
programs are difficult to sustain due to employee turnover, 
lack of recognition of credit for the courses by other higher 


education institutions, changes in the individual's education 
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goals, and declining interest in the specific programs 
offered. 

The issue of "cheap degrees" offered at military bases 
was addressed by Gene Sherron, an American Council on 
Education Fellow [Ref. 30]. He indicates that most of the 
programs are so poor that these programs would be classed 
aS diploma mills if given close scrutiny by the educational 
community. At least 19 colleges and universities from as 
far away as California offer graduate degree programs in 
Washington, D.C. Some major institutions refuse to partici- 
pate because it is their position that quality graduate 
education belongs on campus where students have access to 
full time professors and research facilities. 

So far the accrediting bodies and associations of higher 
education have not forced a policing or correction in the 
proliferation of advanced degrees around the nation. The 
declining enrollments and reduced college budgets have 
accelerated many institutions' quest for the available 
education dollars in the after-work-hours market. The DOD 
1s partially to blame for allowing these institutions to 
offer their educational wares at military installations 
without any control of quality or admissions criteria. The 
local government administrator normally does not have an 
advanced degree and is under intense pressure to keep 
enrollments up and expenses down. Most programs use 


part-time instructors recruited locally. 
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D. THE CURRICULUM PROPOSALS 


1. Naval Postgraduate School Informal Proposals 
In the past seven years the NPS faculty have held 


informal discussions with T&E field activity representatives 
regarding the design of a T&E curriculum. ALeEnough: Te would 
be desireable to have a six quarter program, field activity 
management is strongly opposed to providing full-time 
training for more than one year. The military probably 
would not have difficulty with this point, since most of the 
sponsored NPS curricula are six quarters and longer. 

Table I summarizes informal NPS past proposals for a T&E 
curriculum. These proposals have a high probability and statis- 
tics content and a strong operations research orientation. 
The thesis requirement creates a heavy workload for the 

four quarter curriculum. Since the students most probably 
will have written technical reports prior to attending the 
NPS and will write technical reports upon return to their 
work place, the thesis requirement could be waived without 
reducing the quality of the graduate education. The NPS 
retains the control of the quality of the graduate education 
via having the student in full time residence at NPS; and 
the test activity would reinforce the quality of the educa- 
tional experience through ensuring that the graduate has the 
opportunity to write a formal technical report upon 


completion of the NPS academics. 
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The NPS "strawman" curricula are informal proposals and 
do not really address responsibility for the program. In 
order to ensure that there is a linking of engineering, 
management, and operations research disciplines, some 
thought should be given by the NPS administration to consider- 
ing this curriculum as a Masters of Science in Engineering- 
Management degree program. Although there appears to be 
adequate field activity interest in a T&E oriented curriculum 
at NPS, it has not happended for numerous reasons. Among 
these is a lack of high level Department of the Navy interest 
in a T&E program and the availability of the resources 
required by NPS to implement the program. 

2. Pacific Missile Test Center 

The Pacific Missile Test Center (PMTC) Point Mugu, 
California has advocated the establishment of an advanced 
degree in T&E since the mid 1970's. In 1974 it was able to 
establish an innovative, off-campus graduate education pro- 
gram in engineering with the cooperation of the California 
Seaee University, Northridge [Ref. 31,32]. This program, 
Technical Professional Program, was developed around the new 
engineering graduate employeed by PMTC. 

The objectives were to prepare the engineer for roles 
as a technical specialist in T&E and to develop the engineer's 
skill and competence to plan and analyze large scale tests. 
The students attended classes half-time and worked half-time. 
They were expected to complete the program in one calendar 


year. The work projects were designed to support the 
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classroom studies. Certainly, this was a bold attempt to 
bring graduate education into the real world to support the 
needs of the practioner. Some of the weak points of this 
program appear to be the employee counselor, and a work 
dossier which was maintained by the student and graded as 
satisfying course requirements. This program placed much 
responsibility on the PMTC employees to act as pseudo- 
educators and to ensure that the students had project exper- 
lence equivalent to an on-campus graduate level learning 
Situation. The Technical Professional Program was conducted 
for one year ending in 1976. 

The PMTC has drawn from the experience gained from the 
Technical Professional Program and has proposed the Test and 
Evaluation Intern Program [Ref. 33]. This is a two year 
program designed to develop a T&E Professional Engineer. 
Academic instruction is provided by California State 
University, Northridge and University of California, Santa 
Barbara. The program participants would spend the six 
semesters of the program at various test sites performing 
assignments in T&E of components, subsystems, and systems. 
Formal course work would be attended during part of the 
work day. The courses are listed in Table II. This 
program requires substantial support from the Navy T&E 
Semunecy tol progress. It is a commendable effort because 


it, like the predecessor program, Technical Professional 
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Program, requires close working cooperation between the 
educational institution, the student, and the parciciroeacing 
T&E activity to enrich the learning experience of graduate 


education. 


E. CHARACTERISTICS OF PROFESSIONS 

DOD high level officials have been reluctant to acknow- 
ledge the existence of T&E as being a distinct technical 
discipline. This is understandable because most DOD personnel 
ane not familiar with the T&E process and its role in the 
acquisition of weapon systems. The President's Blue Ribbon 
Panel Report on OT&E supports this observation [Ref. 34]. 
Government personnel, primarily in the development test 
activities, have been the primary spokesmen for calling 
attention to the evolution of a T&E engineering discipline. 
Although they have asked academia to develop formal courses 
and curriculum to support their needs, engineering institutions 
have paid little attention to the requests of the T&E community. 
A review of the characteristics of professions is in order to 
help explain the lack of recognition of T&E as an engineering 
discipline. 

The difference between occupations and professions can be 
viewed as a continuum. The occupations, such as carpentry and 
brick-laying, are on the lower end and medicine and law on the 
upper end of the continuum. A number of attributes have been 


defined which are normally associated with a profession 
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[Ref. 35]. These are: (1) a systematic body of theory; (2) 
professional authority; (3) sanctions by the community; (4) 

a regulative code of ethics: and (5) a professional culture. 
The U. S. Census Bureau in its occupational classification 
includes engineers in the professional category. Again, look- 
ing at the continuum of professions, the engineering profes- 
Sion would be considered below the law and medical professions. 
The environment in which most engineers work differs from 
other professionals [Ref. 25]. The T&E engineers, for example, 
differ as follows: (1) they do not choose their clients; (2) 
they do not have on the job autonomy; (3) they do not have to 
obtain a recognized postgraduate educational certification; 

and (5) they do not have a strong, professionally-backed set 
of standards and ethics. 

The profession which has really arrived is noted by certain 
characteristics which enables it to exercise social control 
Over its community. What evolves then is the professional 
community within the larger social community [Ref. 35]. The 
professional community members tend to take on the following 
characteristics: (1) bound by sense of identity; (2) once in 
the profession, continue for their whole career; (3) share 
values in common; (4) role definitions which are agreed to by 
both members and non-members; and (5) take on a common language 
which is only partially understood by non-members. In addition 
the community has power over its members; its imposed social 


limits are reasonably clear; and it controls the composition 
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of the community in the future through selection of profes- 
Sional trainees and the training processes. The T&E communi- 
ty has not demonstrated that it has moved to the point where 
it exhibits any of these characteristics. 

The knowledge explosion has created specialization within 
Ehe professions. Technology appears to be increasing almost 
exponentially creating a broad technical base and many combina- 
tions of knowledge. Professional specialization is partly 
responsible for creating the technology expansion which in 
turn has increased the amount of specialization. The 
Specialist, for example the electronics engineer, may become 
obsolete because his specialty may still change too fast for 
an individual to keep current. His specialty might be super- 
seded by advances in Knowledge that make his knowledge or 
skill unnecessary or inadequate. To an increasing degree a 
professional career must be regarded as a continuous learning 
experience. The call by the DOD T&E community, for recogni- 
tion as a unique engineering specialty, is an expression of 


this phenomenon. 


FF. FOCAL POINT FOR T&E EDUCATION? 

The DOD sponsors the Defense Systems Management College 
(DSMC), Fort Belvoir, Virginia to train prospective program 
managers. The DSMC primarily provides training to military 
and civilian employees of the DOD who will be employed in the 
acquisition of weapon systems. There is also limited participa- 


tion by contractors. The DSMC, therefore, has become a focal 
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point for the discussion, disemmination, and training in 
program management principles. Problems with the DOD acquisi- 
tion cycle are aired here and possible solutions are discussed 
in a classroom environment. Students are instructed in 
problems and future trends in program management. 

High level DOD management officials participate in DSMC 
Sponsored programs and publications. Thus, through this 
focal point improvements in DOD management techniques can be 
proposed, assessed, and discussed openly. The resulting 
benefits to DOD, by providing program management training to 
military, civilian and contractor personnel are: (1) provides 
a common ground to communicate the DOD policy on acquisition 
management techniques; (2) prepares high level military and 
Civilian personnel who will be assigned to program offices 
for the rigors of program management; (3) introduces these 
personnel to the political aspects of the acquisition of 
weapon systems; and (4) sharing of lessons learned from case 
studies of previous acquisition programs. 

The T&E community has no counterpart to the DSMC. T&E 
has not been institutionalized in DOD. There is no recognized 
focal point in the T&E community where the Services can 
exchange lessons learned, share methodologies and techniques, 
build a knowledge base, disseminate information, conduct 
research, and encourage cooperation. With minimal interference 
from DOD each of the Services have developed separate approach- 


es to TSE. The Joint Logistics Commanders work to improve 
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the participation of each of the Services in Joint service 
acquisition programs. Ad hoc working groups are formed to 
study specific problems and to improve the conduct of i epuigne 
Service programs. The Range Commanders Council, charred by 
PiemDrrector of Defense TSE, occasionally sponsors ad hoc 
working groups to address T&E issues. Only the T&E activities 
designated as part of the Major Range and Test Facility Base 
(MRTFB) participate in this forum. They are listed in 
Appendix A, Table I. 

The DOD through the DDT&SE should sponsor a study or task 
the Services to compile the available knowledge on TSE. It 
should inventory the military and civilian skills required to 


conduct T&E. 


G. TAILORING T&E EDUCATION 

There is increaSing evidence that greater cooperation 
between academic institutions and the worksite for graduate 
education can enrich the learning experience for all the 
participants [Refs. 33,37]. Academia receives immediate feed- 
back for an evaluation of the curriculum and the needs of the 
student relative to the marketplace. The T&E education needs 
are a variable as shown in Figure 1. The more senior the 
DOD official involved in T&E the less are his T&E education 
requirements; and, therefore, T&E seminars of a few days would 
be appropriate. 

The new engineer graduates have been high in analytical 


skills and low in synthesizing and design skills. The early 
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T&E experience is gained through assisting more experienced 
engineers. Through a combination of employer needs and the 
engineer's interests a degree of specialization evolves. The 
engineer, for example, after involvement in several test 
projects with a given type aircraft to determine its per- 
formance characteristics, may become aligned with one 
specialty of aerodynamics. He will then pursue technical 
updating in expectation of receiving a graduate degree. Most 
data shows engineers to be stable and 90% are expected to 
Gegetmiein their field for five years after graduation [Ref. 
25]. Provided the engineer stays at a DOD T&E activity, he 
will observe that more of his time is occupied with manage- 
ment related matters. This situation also begins to surface 
the deficiencies in the engineering education for dealing 
with the social and human problems which are more difficult 
to bound than analytical problems. At this point a decision 
is required on pursuing technical management. Leaving the 
T&E activity, pursuing a technical specialist career or 
developing managerial skills are the main options available. 
Some T&E activities have been more successful than others in 
developing a dual career advancement program; however, this 
Still remains a critical decision point for most engineers. 
The evolution of education needs as shown in Figure 1 gives 
evidence of the need for flexibility in the concept of T&E 
education. The needs of the specific T&E activity should be 


matched with the needs of the individual to perform his duties. 
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It is at this point T&E education can have a recognizeable 
form. The types of education which are shown in Figure 2 
depict in general the needs of T&E personnel. The point at 
which these education needs change are peculiar to each 
individual and T&E activity. It does provide, however, an 
indication of the range of education required. The technical 
report writing and test planning need to be reinforced and 
integrated into the specific technical education courses. 

The repetition and reinforcement of these basic skills are 
critical in communicating T&E results concisely. 

In the case of the military officer entering T&E of air- 
craft weapons systems a different education evolution occurs. 
The operating skills of the officer are used to support the 
T&SE process. For example, it depends on which T&E activity 
he is assigned to as to whether or not he will receive 
specific training at the Naval Test Pilot School. This 
training develops the skills to conduct engineering flight 
testing. The military officer T&E education needs are 
normally different than those for the engineer. He may only 
spend two years in TSE and is more concerned with operating 
skills and management education. He may have already attended 
the Naval Postgraduate School and received graduate education 
in engineering prior to coming to the T&E activity. The 
officer who has a designated subspecialty as an aviation 
engineering duty officer normally has an advanced technical 


degree prior to coming to the T&E activity. His education 
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needs would lie more to the right in Figures 1 and 2. If a 
T&E curriculum were available, for example at the NPS, then 
the officers completing the curriculum would be more effec- 
tive at their jobs sooner and require less on-the-job train- 
ing time. The officers interviewed in reference [2] indicated 
that knowledge about T&E principles and program management 
would have helped their effectiveness in their jobs and 
reduced their on-the-job training. 

The need for T&E education so far has come from the 
personnel assigned to the T&E activities. The T&E body of 
knowledge is disbursed throughout the DOD T&E activities. 

The Director of Defense T&E (DDT&E) support is needed for the 
Support of the activities’ efforts to define their graduate 
education requirements. DDT&E could contract for a study 
with industry, a nongovernment academic institution, or a 
military postgraduate school on the issues involving T&E 
education. Some of the parameters this study could consider 
are: (1) the bounds and content of the T&E body of knowledge; 
(2) a focal point for T&E knowledge; (3) T&E education; (4) 
military careers in T&E; (5) future trends in T&E; and (6) 
rotation of civilian and military personnel between T&E 
activities. 

The Congress has not always been pleased with the Services’ 
management of T&E. Congress would be supportive of DDT&E 
efforts to exercise management improvement of the T&E process. 


Congress has expressed concerns in the past that the Service 
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filter T&E results of major weapon systems in order to enhance 
the chances Congress will provide funding for procurement. 
Various studies have critisized the delays, duplications, and 
cost of T&E. These are symptoms which have been treated by 
numerous directives and instructions and even establishment 

of the DDT&SE. Possibly, through T&E education, developing a 
body of T&E knowledge, and establishing a DOD focal point for 
distribution of this knowledge the significance and role of 


T&E can be fairly assessed. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 


1. General 


The hypothesis is not presently supportable. There 
is little evidence to justify the establishment of a graduate 
T&E curriculum. There is a real need, however, for a compila- 
tion of the information available on T&E in the DOD. It is 
the DDT&E responsibility to define, support, and initiate 
fers effort. 

ees OeC tlic 

The DOD officials, in attempting to solve the problems 
of the acquisition of major weapon systems, identified weak- 
nesses in the role of T&E. The solutions have been to change 
organizational structure and to change the timing and participa- 
tion of T&E personnel in the acquisition cycle. The problems 
in T&E, however, seem more fundamental. The area of T&E 
education has not been considered by high level DOD officials 
BoeGelewamee tO improving the effectiveness of T&E. 

The Services have developed different implementations 
of the concept of independent testing as required by DOD 
directives. These differences raise policy, procedural and 
philosophical issues, when the Services are required to par- 
ticipate in joint-service acquisition programs. The same sort 


of situation also exists in combined intra-Service testing. 
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The trend for the future, in order to decrease duplication 

of efforts and acquisition costs and schedules, is to increase 
the number of joint-Service programs and of intra-Service 
combined tests. This points to a need for increased high 
level DOD officials’ attention to the education and training 
needs of T&E personnel to reduce communication problems. 

It 1s not generally accepted or recognized by high 
level DOD officials that T&E requires unique education. T&E 
in support of the acquisition cycle is, however, unique to 
DOD. The body of knowledge, professional society representa- 
tion, and community characteristics do not exist in a highly 
visible form. The DOD T&E community is not well organized 
or well represented. There is no consensus on what a T&E 
specialist is. Most engineers and officers would not, if 
asked, claim to be a T&E specialist but would instead identify 
with the engineering discipline in which they earned their 
degree, for example electrical engineering. 

Academic institutions have not generally embraced the 
concept of T&E as a specialty. There has been little academic 
interest in establishing T&E courses or developing faculty to 
support a T&E curriculum. The educational parameters necessary 
to support T&E seem to point more toward a blend of engineering 
and management education. 

The TSE personnel are not very mobile. There has been 
very little effort in DOD to utilize T&E personnel across 


Service lines or intra-Service between test agencies. The 
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opportunities, therefore, for cross-pollenization of T&E 
skills, policies, test methods and lessons learned have been 
minimal. Government personnel policies do not provide for 
the transfer of T&E personnel to programs where their talents 
aoe Most needed. The evolution and identity of the T&E 
Specialists with special education requirements can only be 
impeded by this policy. 

The Naval Aviation Executive Institute (NAEI) program 
in cooperation with the Naval Postgraduate School could be 
the leader in the DOD for defining and supporting the graduate 
education needs of T&E personnel. Increased cooperation and 
flexibility are required between the NAEI, NPS, and the Navy's 
T&E agencies. The uncertainties, of future civilian T&E 
personnel participation in NPS provided education, could be 
resolved through improved communications of the needs of each 


Pemenowparticipating organizations. 


B. RECOMMENDATIONS 
1. The Director Defense T&E (DDT&E) 
The DDT&E should: 
(1) establish a focal point for the compilation and 
dissemination of T&E information. 
(2) initiate a study to define the DOD TS&E body of 
knowledge. 


(3) sponsor an annual seminar on DOD T&E. 
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(4) develop a seminar course consisting of 20-40 hours 
OL Anstruction on DOD Ts for high level BOD 
Officials. 
(5) assess the T&E related education needs of the 
T&E agencies and prepare a plan to support these 
eee 
2.) Lhe Services 
Fach of the Services should: 
(1) compile its existing body of T&E knowledge. 
(2) review the educational needs of its T&E personnel. 
(3) define career paths for T&E personnel which includes 
involvement in acquisition management decision 
making. 
3. NPS, NAEI, and the Navy's RDT&E Agencies 
Representatives from these activities should meet to 
review the trends in graduate education, and the need to 
involve the NPS more closely in the integration of education 


with the education needs of the T&E practioners. 
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APPENDIX A 


OMB CIRCULAR 
A-109 
Major System Acquisitions 


DOD 5000.1 DOD 5000.2 DOD 5000.3 | 
Major System Major System Test & Evaluation | 


Acquisition Acquisition Procedures 





OPNAVINST 3960.LOA 
Test and Evaluation 





NAVMATINST 3960.6A 
Test and Evaluation 


NAVAIRINST 3960.2A 
Test and Evaluation 





v 
TEMP 


(Test and Evaluation Master Plan) 


Figure 2. The Flow of Instructions to Develop a TEMP. 
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APPENDIX A 


TABLE I 


DEPARTMENT OF DEFENSE MAJOR RANGE 
AND TEST FACILITY BASES 


ARMY 


Cold Regions Test Center 
Fort Greely, Alaska 


Tropic Test Center 
Bourke Clayton, Canal Zone 


Yuma Proving Ground 
Yuma, Arizona 


Jefferson Proving Ground 
Madison, Indiana 


NAVY 


Pacific Missile Test Center 
Point Mugu, California 


Atlantic Undersea T&E 
Center 
Andros Island, Bahamas 


Naval Air Test Center 
Patuxent River, Maryland 


Naval Air Propulsion Test 
Se yneeie 


Trenton, New Jersey 


ATR FORCE 


Space & Missile Test Center 
Vandenberg AFB, California 


Eastern Test Range 
Patrick AFB, Florida 


White Sands Missile Range 
New Mexico 


Kwajalein Missile Range 
Marshall Islands, Pacific 


Electronic Proving Ground 
Fort Huachuca, Arizona 


Dugway Proving Ground 
Salt Lake City, Utah 


Aberdeen Proving Ground 
Aberdeen, Maryland 


Naval Air Engineering Center 
Lakehurst, New Jersey 


Naval Weapons Center 
China Lake, California 


Atlantic Fleet Weapons Training 
Facility 


Roosevelt Roads, Puerto Rico 


Flight Test Center 
Edwards AFB, California 


Armament Development & Test 
Center 
Eglin AFB, Florida 
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AIR FORCE (Continued) 


Satellite Control Facility Air Defense Weapons Center 


Sunnyvale, California Pyneall APB, Florida 

Tactical Fighter Weapons Arnold Engineering Development 
Center Center 

Nellis AFB, Nevada Tullahoma, Tennessee 


4950th Test Wing 
Wright-Patterson AFB, Ohio 
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